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About This Research Brief 

This research brief is part of the American Institutes for Research’s continuing effort to make research relevant 
to policymakers and practitioners. Our mission is to conduct and apply the best behavioral and social science 
research and evaluation toward improving peoples’ lives, with a special emphasis on the disadvantaged. This 
brief highlights the findings of a rigorous, federally funded three-year randomized control trial examining the 
impact of an online Algebra I course on eighth-grade students. 

Who Conducted the Original Study? 

The Access to Online Algebra study was conducted by the American Institutes for Research (AIR) and the Education 
Development Center, Inc. (EDC) for the 2006-2001 Regional Educational Laboratory Northeast and Islands 
(www.relnei.org/). Authors of the study’s final report are Jessica Heppen, AIR; Kirk Walters, AIR; Margaret 
Clements, EDC; Ann-Marie Faria, AIR; Cheryl Tobey, EDC; Nicholas Sorensen, AIR; and Katherine Culp, EDC. 
The report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute 
of Education Sciences, under contract ED-06C0-0025. 

The citation for the final report is: 

HeppenJ.B., Walters, K., Clements, M., Faria, A., Tobey, C., Sorensen, N., and Culp, K. (201 2). Access to Algebra 
I: The Effects of Online Mathematics for Grade 8 Students. (NCEE 201 2-4021 ). Washington, DC: National Center for 
Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. 

For the full final report, visit: http://ies.ed.gov/ncee/edlabs/regions/ northeast/ pdf/ REL_2 01 24021 .pdf 
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Introduction 


Online courses are increasingly seen as a viable way 
to expand the number and types of courses offered 
and broadening access to key courses, particularly in 
small and rural schools. 1 Algebra I in middle school 
exemplifies an important course that educators and 
policymakers want to be more available to students. 

To study the effectiveness of using an online course 
to broaden eighth graders’ access to Algebra I, the 
American Institutes for Research (AIR) and the 
Education Development Center, Inc. (EDC) performed 
the first rigorous evaluation of the impact of an online 
course on student outcomes. This three-year study 
examined the effects of offering a fully online Algebra I 
course to eighth-grade students in mostly rural schools 
in Maine and Vermont. 
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Key Findings 


The study found that offering Algebra I as an online 
course is an effective way to broaden access for 
students in schools where access to the course is 
typically limited. Specifically: 

■ For students whose schools deemed them eligible 
for Algebra I in eighth grade, taking the course: 

■ Improved their algebra achievement at the end 
of eighth grade, and 

■ Doubled their chances of taking an advanced 
math course sequence in high school. 

■ Taking the online course (instead of the usual 
eighth-grade math class) had no negative effect 
on students’ general math achievement at the end 
of eighth grade. 

■ Removing eligible students from the general math 
class to offer them the online Algebra I course had 

no discernible side effects on now-eligible students’ 
achievement or course-taking outcomes. 

This brief describes the study’s purpose, methods, 
findings, and implications. 
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Study Rationale 
and Design 


In 1997, the U.S. Department of Education asserted 
that all states should invest in expanding access to 
Algebra I for middle school students. This sentiment 
was echoed in 2008 by the National Mathematics 
Advisory Panel, which recommended that “all prepared 
students [should] have access to an authentic algebra 
course — and [that districts] should prepare more 
students than at present to enroll in such a course 
by Grade 8.” These policy statements are based on 
research that shows that Algebra I is a gateway to more 
advanced courses in high school and college and that 
students who succeed in Algebra I in middle school 
have more success in math throughout high school 
and college than students who take Algebra I later." 

With the push to expand access to Algebra I in middle 
grades, national grade-8 Algebra I enrollments have 
increased from 1 6 percent in the 1 990s to 31 percent 
in 2007."' In general, students who take Algebra I in 
(or before) eighth grade are higher-achieving students 
than those who take the course in high school. 1 '' 


However, not all high-achieving students have the 
opportunity to take Algebra I in or by grade 8. As 
of 2009, about one-quarter of the highest-achieving 
students did not take a formal Algebra I course in 
middle school. v In rural schools, nearly 40 percent of 
high-achieving students did not take Algebra I in grade 
8. In suburban and urban schools, by comparison, 
the proportions were close to the national average of 
about 25 percent. 

Figure 1: Percentage of high achievers 
(as of 5th grade) that did NOT take 
Algebra I in 8th Grade as of 2009 
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Source: NCES Early Childhood Longitudinal Study — Kindergarten Cohort 
(ECLS-K) K-8 Full Sample Public-use Data File (U.S. Dept of Education, 2009). 
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Presumably, many of the students who miss out attend 
schools that do not offer the course or where access 
is limited — a particular issue in rural schools. As of 
2009, nearly one-quarter (24 percent) of rural schools 
did not offer Algebra I in grade 8, higher than the 
percentage of urban schools (21 percent) and 
suburban schools (9 percent) that did not offer the 
course. 


Figure 2: Percentage of schools that 
reported they did NOT offer Algebra I 
to 8th-grade students 
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Source: NCES Early Childhood Longitudinal Study — Kindergarten Cohort 
(ECLS-K) K-8 Full Sample Public-use Data File (U.S. Dept of Education, 2009). 
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Measuring the Impact of Online Algebra 

To measure the overall impact of offering online 
Algebra I to eighth graders, the study team randomly 
assigned eligible and willing schools either to offer 
the online course to students they considered ready, 
or to offer their “business-as-usual” math curriculum 
(described below). This strong and simple research 
design allowed the research team to examine 
the potential benefits and possible unintended 
consequences of implementing the online course. 


Measuring Potential Benefits 

Sixty-eight middle schools in Maine and Vermont 
participated. They either did not offer Algebra I to 
eighth graders at all, or for various reasons they 
offered the course to only some of their algebra-ready 
students. 


Definition of “Algebra-Ready” Students 

Schools decided which students would be eligible for 
the course. That way, the situation reflected a “real- 
world” setting. Schools determined students’ eligibility 
based on teacher recommendations, grades in sixth 
and seventh grade math courses, prior test scores, 
and related factors. Eligible students were identified 
before schools were randomly assigned to offer the 
online course. 


Study Design — Random Assignment of 
Schools to Offer Online Algebra I or Not 

The 68 schools were randomly assigned to one of two 
groups. Halfofthe schools received an online Algebra 
I course (at no cost) to offer during the 2008-09 
school year. The “control schools” offered their usual 
mathematics curriculum. 

The algebra-ready students are represented in the top 
boxes in the figure below. To measure the potential 
benefits of online Algebra I, algebra-ready students 
in treatment schools were compared to algebra-ready 
students in control schools (arrow A). 

Figure 3: Measuring potential benefits of 
online Algebra I 
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The rest of the eighth graders in the school were the 
“non- algebra-ready” students, represented in the lower 
boxes. These students were included in the study to test 
for the possible unintended consequences of offering 
an online course to eligible students. The study did 
not compare outcomes for algebra-ready students 
vs. non-algebra-ready students, but it did compare 
their baseline characteristics. Students identified by 
schools as eligible for Algebra I in grade 8 were, in 
fact, significantly higher achieving than students who 
were not identified as algebra-ready. 

Why Measure Potential Unintended 
Consequences? 

Offering an online Algebra I course in schools that do 
not typically offer the course represents a clear change 
to the business-as-usual eighth-grade mathematics 
program. Therefore, the study also examined whether 
there are potential unintended consequences of 
offering online Algebra I to algebra-ready students — 
for them or their non-algebra-ready peers. 

For example, taking online Algebra I instead of the 
general eighth-grade math class could adversely affect 
algebra-ready students’ general math achievement (as 
measured, for example, on statewide assessments). 

Moreover, providing online Algebra I to algebra-ready 
students could have unintended consequences for 
the non- algebra-ready students in the school. These 
students remain in the general grade-8 math class. 
Various changes could occur in the general math class 
when the algebra-ready students are removed to take 
online Algebra I — such as smaller class sizes, a change 
in course emphasis (for example, less algebra), and 
more homogenous and lower ability levels among the 
students. 


Schools and Students in the Study 

Most of the schools (61 out of 68) in the study were 
classified as rural, and nearly all (94 percent) ofthem 
were Title I schools. In 76 percent of the schools, 
the grade span was l<-8 or pre-l<-8. As of the 2007- 
OS school year, average total school enrollment for 
the participating schools was 186 students and the 
student population was 95 percent white, with nearly 
half (48 percent) eligible for a free or reduced-price 
lunch. Slightly more than half (53 percent) of the 
students in the participating schools had scored at or 
above proficiency on the state math assessment the 
prior year. 

The average number of eighth graders in each school 
was 32, and more than half of the schools had 1 6 or 
fewer students in that grade. 

On average, the schools identified 6.5 students 
per school as algebra-ready — roughly 22 percent 
of their eighth graders. The next table summarizes 
participating students’ background characteristics. 

Table 1: Participating students’ background 
characteristics 



a State assessment scores were standardized because the two states used different 
assessments. Standardized scores have a mean ofzero and a standard deviation of 1 . 
Algebra-ready students scored, on average, 1 standard deviation above the average 
relative to all students in the sample. Non-algebra-ready students scored, on average, 
one-quarter of a standard deviation below the average. 
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Of the 440 algebra-ready students, 218 attended 
schools assigned to the “treatment” group and 
were offered the option of taking the online Algebra 
I course. Nearly all of them — 97 percent (211 
students)— enrolled. A total of 222 algebra-ready 
students attended the control schools and took a 
variety of math classes, which are described below. 

Ofthe 1 ,445 non-algebra-ready students, 744 attended 
treatment schools and 701 attended control schools. 
Analysis of baseline characteristics confirmed that the 
random assignment process produced treatment and 
control groups that were not significantly different 
from each other at the outset ofthe study. 

Description of the Online Algebra I 
Course 

The online course used in the study is a completely 
Web-based course offered by Class.com, based in 
Lincoln, Nebraska. The course was one of Class, 
corn’s off-the-shelf products. It had three instructional 
components: the online course software, an online 
teacher (provided by Class.com), and an on-site 
proctor (provided by the school). 

Analysis ofthe course content determined that the 
topics covered in Class. corn’s course were similar to 
those in textbooks typically used in the region. The 
material for each topic was presented as an electronic, 
interactive textbook that included computerized 
direct instruction, guided practice (“your-turn” 
problems) and practice problem sets (both with 
automated feedback), and quizzes and exams that 
provided immediate scores. Other activities included 
demonstrations of content materials, audio clips, 
interactive applets that present questions and guided 
solutions, a messaging feature that allowed students 
to exchange messages with the online teacher, and 
a discussion board to which students could post 
questions and comments. 


Students taking the online course were assigned 
to a specific section and taught by a teacher hired, 
trained and supervised by Class.com. The teacher was 
responsible for providing instruction and supporting 
student learning. 

There were a total of 1 1 course sections across the 
treatment schools, taught by a total of 8 online 
teachers. The total number of students enrolled in the 
online course was 242. 1 

Schools had to provide a staff member to serve as an 
on-site proctor. The proctor did not have to be a math 
teacher and was not required to provide instruction. 
Proctors were expected to supervise behavior, serve 
as a contact person for students and parents, oversee 
quizzes and exams, and act as a liaison between the 
online teacher and the school, students and parents. 


1 This included the 211 algebra-ready students already mentioned plus 
another 31 non-algebra-ready students who were enrolled in the course in 
fall 2008. Schools were allowed to enroll additional students who had not 
been on their initial list of algebra-ready students in spring 2008. However, 
students who were not on those lists were considered non-algebra-ready 
for the purpose of all study analyses. 
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Implementation 

Findings 


This description of the implementation of online 
Algebra is based on the analysis of data from the online 
course management system and proctors’ weekly logs. 
Key findings include: 

■ Most schools opted not to place students taking the 
online course in a separate space or at a different 
time from the regular eighth-grade math class. 

■ In most (80 percent) of the treatment schools, 
the regular eighth-grade math teacher served 
as the proctor. 

■ In 69 percent of the schools, students taking 
the online course sat in the same classroom 
as students taking the regular eighth-grade 
math class. 

■ The types and amount of interaction between 
students taking the online course and their 
teachers and proctors deviated sharply from initial 
expectations. 

■ Online teachers spent less time communicating 
directly with students than expected. The 

online teachers logged in to the course at least 
once a day to monitor students’ activity or 
progress, but they communicated directly with 
only about 25 percent of the students every day. 


Communications containing math content were 
infrequent; however, when a student contacted 
the online teacher directly, the teacher almost 
always (96 percent of the time) replied within 
24 hours. 

■ In-class proctors spent more time providing 
math content support than expected. Although 
the proctors’ role did not require providing math 
instruction, they spent an average of about 60- 
75 minutes per week answering algebra-related 
or other non-algebra-related math questions for 
students taking the online course. 

■ Rates of course completion in the online course 

varied. 

■ The average number of course units completed 
was 7.5, or 85 percent of the online course. 

■ 43 percent of the online algebra students 
completed the entire course, including all 9 core 
units of the full-year course (5 units in Algebra 
IA and 4 in IB). 

■ 82 percent of the online algebra students 
completed the first half of the course (Algebra 
I A) and part of Algebra IB. 
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Content of the Business-as-Usual Math 
Classes - to What Was Online Algebra 
Compared? 

At the outset of the study it was known that grade-8 
students in schools with no or limited access to a 
formal Algebra I course are still exposed to algebraic 
content. Therefore, this study was not a comparison of 
online algebra with no algebra whatsoever. To clarify 
the true nature of the contrast between the treatment 
and control conditions, the study team closely 
documented the mathematics offered to students in 
control schools. 

The study found that most (80 percent) ofthe algebra- 
ready students in control schools took a general 
grade-8 math class. To understand the content ofthe 
math classes offered, the study team analyzed course 
materials and gauged the extent to which the topics 
covered are typically found in formal Algebra I courses. 

■ In nearly half of the control schools, the general 
math class focused on algebraic content 75 percent 
or more ofthe time, and nearly all ofthe classes 
(94 percent) had a curricular focus on algebraic 
content of half or more. Thus, algebra-ready (and 
non-algebra-ready) students in control schools 
were exposed to a substantial amount of algebra 
content, even though they had little or no access 
to a formal Algebra I course. 

■ Moreover, 45 students — 20 percent ofthe algebra- 
ready students in control schools — took a separate 
Algebra I course taught at the local high school or 
middle school. 
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Study Results: 
Impacts of 
Online Algebra I 


Potential Benefits for Algebra-Ready 
Students 

The study measured potential benefits for algebra- 
ready students for two main outcomes: 

■ Algebra knowledge and skills, measured with 
an end-of-year computer-adaptive algebra 
assessment taken by all students in participating 
schools. 

■ High school course-taking — specifically, the 
likelihood of participating in an advanced math 
course sequence — based on information about 
ninth-grade courses taken, grades earned, and 
tenth-grade course enrollments.'' 1 

To test whether the online course benefitted 
students’ algebra knowledge and skills and high 
school coursetaking patterns, the study compared 
outcomes for all 218 algebra-ready students in 
treatment schools (who had access to online Algebra 
I) to outcomes for all 222 of their algebra-ready 
counterparts in control schools. The results showed 
that the online course had a significant positive 
impact on both outcomes. 


1. Students in treatment schools (that offered 
online algebra) scored higher than students in 
control schools on an algebra achievement test 
given at the end of eighth grade. The difference 
between students in treatment schools and 
students in control schools was 5.5 points on a 
scale ranging from 400-500 (shown in the graph 
at right). v " This difference is equivalent to moving 
students from the 50th to the 66th percentile in 
achievement by taking the online course. 


Figure 4: Algebra-ready students’ 
end-of-eighth grade algebra scores 
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2. Students from treatment schools (that offered 
online algebra) were more than twice as likely as 
control students to take an advanced math course 
sequence in high school. Using information about 
courses taken, grades earned, and courses planned 
in the first 2 years of high school, we found that the 
chances of being in an advanced course sequence 
were 51 percent for students who attended middle 
schools that offered the online Algebra I course 
versus 26 percent for control students. Vl " 


Figure 5: Algebra-ready students’ 
predicted probability of advanced math 
course-taking 
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Potential Side Effects 

We also found no significant unintended consequences 
as a result of offering online Algebra I to eligible 
students. Specifically: 

1 . Algebra ready students’ general math achievement 
was not affected by taking online Algebra I. 

All students in the participating schools took a 
computer-adaptive general math assessment at 
the end of eighth grade, and scores for algebra- 
ready students in treatment schools did not differ 
significantly from their counterparts’ in control 
schools. 


Figure 6: Algebra-ready students’ end-of- 
eighth grade general math scores 



Treatment (N=21 8) Control (N=222) 


This important finding responds to potential concerns 
that students who take Algebra I in eighth grade — in 
this case as an online course — will miss out on general 
math content. Eighth grade state math assessments 
typically cover a range of topics, and some schools 
may worry that offering online Algebra I will lead to 
lower proficiency rates. The study findings suggest that 
students who take online Algebra I in eighth grade do 
not score significantly differently on a general math 
assessment than they would have had their schools 
not offered the online course. 

2. Non-algebra-ready students were not affected 
by their schools’ adoption of the online Algebra 
I course on any of the outcomes AIR examined. 

These included their algebra and general math 
scores and their projected high school course- 
taking. 

Summary of Findings 

This study demonstrated that, compared with 
business-as-usual math instruction, students with 
access to online Algebra I in eighth grade learned more 
algebra that year and later doubled their chances of 
taking advanced math courses in high school. These 
results show that online Algebra I offered in grade 8 
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benefits students by boosting algebra knowledge and 
skills and by opening doors to more advanced course 
opportunities in later years. 

The study also found that there were no negative 
side effects on students in schools that adopted 
online Algebra I. Such potential consequences of 
implementing an online course were important to 
educators in the types of schools that participated in 
the study, and for schools around the country. When 
a school decides to use an online course (or any other 
innovation) to create opportunities for some but not 
all students, the school must question whether any of 
those students might be harmed by that choice. 

In combination, the findings lead us to conclude that 
offering online Algebra I is an effective way to broaden 
access to this specific course in eighth grade and, later, 
to more challenging math courses in schools where 
access is typically limited. 
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Conclusions 


Implications 

The study was designed to provide information to 
educators who are looking for ways to offer a key 
gateway course — Algebra I — to their eighth graders 
who are ready for it, but for various reasons cannot 
typically offer full access to the course in a standard or 
traditional way. The goal for offering the online course 
was not only to benefit short-term algebra achievement 
but to influence a sequence of math opportunities and 
outcomes overtime. It may seem obvious that students 
with access to an online Algebra I course should learn 
more algebra and take more advanced courses earlier 
in high school than those who do not. But for multiple 
reasons, the results observed were not necessarily 
obvious and addressed gaps in the research base. 

Before this study, there was no prior evidence that 
an online version of a formal Algebra I course could 
successfully be offered to middle school students in 
terms oftechnology and content support. Additionally, 
though the logistical implementation of the course 
went as planned, just under half (43%) ofthe students 
who enrolled in the course fully completed it, meaning 
that many ofthe algebra-ready students in the 
treatment group were not exposed to the entire course. 
At the same time, their algebra-ready counterparts in 
control schools were exposed to a substantial amount 


ofalgebraic content in the context ofthe general math 
classes, and 20 percent of them actually did take a 
formal Algebra I course. Despite these circumstances, 
the results demonstrate that online Algebra I, as 
implemented in the study, was more effective for 
promoting students’ success in math than existing 
practices in these schools. 

Limitations and Future Directions 

The online course chosen, Class. corn’s Algebra I 
course, was similar in content, focus and structure 
to the offerings of other providers. However, it is not 
clear that results would have been similar if another 
provider had been chosen. Moreover, the results 
observed in this study cannot necessarily be generalized 
to more recently developed online courses, or to 
courses in other subject areas. 

As the use of online courses continues to increase in 
U.S. schools, researchers should continue to study 
the effects on student achievement and course-taking 
patterns in key content areas. Further research on 
online courses should compare online options to 
various other relevant business-as-usual situations 
and curricula. These include school settings in which 
students lack access to specific courses (where the 
control group does not take the course) and in which 
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particular courses are oversubscribed (where the 
control group would take a standard face-to-face 
version of the online course). 

Schools around the country will continue to search for 
innovative ways to expand their course offerings, and 
online courses are an increasingly available option. 
This study provides early evidence that under particular 
conditions, students benefit when schools expand 
access to Algebra I by offering it online. 

For more information about the study, please visit: 
http://www.air.org/focus-area/ edu cat ion/ index. 
cfm?fa=viewContent&content_id=1 641 &id = 1 0 
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Endnotes 


i Schwartzbeck et al. 2003; Hanum et al. 2009. 

ii Stevenson, Schiller, and Schneider 1 994; Schneider, 
Swanson, and Riegle-Crumb 1 998; U.S. Department 
of Education 1 996; Smith 1 996; Spielhagen 2006. 

iii Loveless 2008. 

iv Walston and Carlivati McCarroll, 2010. 

v Prior research suggests that successful completion 
of Algebra II by the end of tenth grade signals 
participation in an “advanced course sequence” 
(Schneider et al. 1998; Stevensen et al. 1994). 
Based on this research, we coded high school 
course sequences for students in the study as 
“advanced” if the students successfully completed 
a full-year course above Algebra I in ninth grade 
(e.g., Geometry) with a grade of C or higher, and 
enrolled in the next course in the sequence (e.g., 
Algebra II) for grade 10. 

vi This difference is equal to an effect size of 0.40 
standard deviations, considered moderate to large 
for an educational intervention. 


vii Exploratory analyses conducted as part of this 
study tested the effect of online Algebra I in eighth 
grade on the likelihood that students would 
“double up” on full-year math courses in ninth or 
tenth grade (or both). The results showed that 
students from control schools were more than 
twice as likely to double up in math courses in 
grade 9 or 1 0 as students who took online algebra 
in eighth grade. This suggests that in addition to 
affected whether students pursued an advanced 
course sequence, online algebra in grade 8 had 
an impact on how students entered an advanced 
course pathway. 
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